Log File Analysis in Cloud with Apache Hadoop and Apache Spark

Proceedings of: Second International Workshop on Sustainable Ultrascale Computing Systems (NESUS 2015). Krakow (Poland), September 10-11, 2015.Log files are a very important set of data that can lead to useful information through proper analysis. Due to the high production rate and the number of devices and software that generate logs, the use of cloud services for log analysis is almost necessary. This paper reviews the cloud computational framework ApacheTM Hadoop R, highlights the differences and similarities between Hadoop MapReduce and Apache SparkTM and evaluates the performance of them. Log file analysis applications were developed in both frameworks and performed SQL-type queries in real Apache Web Server log files. Various measurements were taken for each application and query with different parameters in order to extract safe conclusions about the performance of the two frameworks.The authors would like to thank Okeanos the GRNET’s cloud service for the valuable resources

Distributed Processing in Cloud Computing

Proceedings of the First PhD Symposium on Sustainable Ultrascale Computing Systems (NESUS PhD 2016) Timisoara, Romania. February 8-11, 2016.Cloud computing offers a wide range of resources and services through the Internet that can been used for various purposes. The rapid growth of cloud computing has exempted many companies and institutions from the burden of maintaining expensive hardware and software infrastructure. With characteristics like high scalability, availability and fault tolerance, cloud computing meet the new era needs for massive data processing at an affordable cost. In our doctoral research we intend to study, analyze, evaluate and make proposals in order to further improve the performance of cloud computing.European Cooperation in Science and Technology. COS

Vaspin: a novel adipokine, member of the family of serine protease inhibitors

In 2000, the novel adipokine vaspin, which belongs to the superfamily of serpins, was isolated from visceral adipose tissue. Vaspin is mainly produced in the visceral adipose tissue and is related to insulin resistance, blood glucose levels, sex hormones (women have higher levels compared to men) and nutritional status. Moreover, vaspin levels are modulated by weight loss and several agents, and it possibly constitutes a connecting link between obesity and its associated metabolic disorders. Many patients with polycystic ovary syndrome have insulin resistance, obesity (mostly visceral) and glucose intolerance, conditions associated with abnormalities in the production of vaspin. The role of vaspin in the regulation of human metabolism is unclear at present, but it appears that vaspin might represent a novel marker of obesity and insulin resistance. However, the controversial findings of existing studies on vaspin stress the need for further research in women with obesity and metabolic disorders in order to elucidate the role of this adipokine in these diseases and particularly in the polycystic ovary syndrome

Programming Languages for Data-Intensive HPC Applications: a Systematic Mapping Study

A major challenge in modelling and simulation is the need to combine expertise in both software technologies and a given scientific domain. When High-Performance Computing (HPC) is required to solve a scientific problem, software development becomes a problematic issue. Considering the complexity of the software for HPC, it is useful to identify programming languages that can be used to alleviate this issue. Because the existing literature on the topic of HPC is very dispersed, we performed a Systematic Mapping Study (SMS) in the context of the European COST Action cHiPSet. This literature study maps characteristics of various programming languages for data-intensive HPC applications, including category, typical user profiles, effectiveness, and type of articles. We organised the SMS in two phases. In the first phase, relevant articles are identified employing an automated keyword-based search in eight digital libraries. This lead to an initial sample of 420 papers, which was then narrowed down in a second phase by human inspection of article abstracts, titles and keywords to 152 relevant articles published in the period 2006–2018. The analysis of these articles enabled us to identify 26 programming languages referred to in 33 of relevant articles. We compared the outcome of the mapping study with results of our questionnaire-based survey that involved 57 HPC experts. The mapping study and the survey revealed that the desired features of programming languages for data-intensive HPC applications are portability, performance and usability. Furthermore, we observed that the majority of the programming languages used in the context of data-intensive HPC applications are text-based general-purpose programming languages. Typically these have a steep learning curve, which makes them difficult to adopt. We believe that the outcome of this study will inspire future research and development in programming languages for data-intensive HPC applications.Additional co-authors: Sabri Pllana, Ana Respício, José Simão, Luís Veiga, Ari Vis

Virtualization technologies and distributed processing in cloud computing

Cloud computing is currently a widespread and commonly used technology. The potentially infinite computing resources of the cloud are accessible to anyone with internet access and do not require any prior investment in hardware or software by the user. There are several reasons, such as the Pay As You Go cost model, the high level of scalability and flexibility, which lead to the increasing adoption of the cloud by large companies and organizations as well as by ordinary users.Since the beginning of cloud computing, virtualization technologies have been an integral part of the cloud. Various virtualization technologies allow multiple users to share existing computing infrastructure simultaneously and, at the same time, virtualization technologies increase resource utilization and reduce overall costs and energy consumption. Virtual Machines (VMs) are traditionally used to allocate computing resources and create isolated environments among users. Recently, a lighter virtualization technology has appeared in the clouds, containers. The main difference between containers and VMs is that containers share the operating system of the host machine, whereas each VM has its own operating system.The rise of cloud computing and distributed processing frameworks has contributed to reduce costs and facilitate the storage and processing of big data. Many cloud platforms further simplify the execution of distributed workloads by offering services, tools and even VM images with pre-installed distributed processing frameworks. Two of the most common distributed processing frameworks, which have been also studied in this thesis, are Hadoop and Spark.This thesis consists of two sections. In the first section, the Hadoop and Spark distributed processing frameworks are studied, researched, and empirically evaluated. In the second section, modern virtualization technologies are presented and researched, their combination is recommended and evaluated and their adoption in latest cloud computing services is proposed and evaluated.Τα υπολογιστικά νέφη είναι πλέον μια ευρέως διαδομένη και συχνά χρησιμοποιούμενη τεχνολογία. Οι δυνητικά άπειροι υπολογιστικοί πόροι των νεφών, είναι προσβάσιμοι σε οποιονδήποτε έχει τη δυνατότητα σύνδεσης στο διαδίκτυο και δεν απαιτούν από τον χρήστη προγενέστερη επένδυση σε υλικό ή λογισμικό. Μερικοί λόγοι που οδηγούν στην όλο και αυξανόμενη υιοθέτησή τους από μεγάλες εταιρίες και οργανισμούς καθώς και από απλούς χρήστες είναι ότι ακολουθούν ένα ευέλικτο μοντέλο χρονοχρέωσης, παρέχουν υψηλό επίπεδο κλιμακωσιμότητας (scalability), ευελιξίας και ελαστικότητας. Από την απαρχή των υπολογιστικών νεφών μέχρι και σήμερα, οι τεχνολογίες εικονικοποίησης αποτελούν αναπόσπαστο συστατικό τους. Οι διάφορες τεχνολογίες εικονικοποίησης επιτρέπουν την ταυτόχρονη διαμοίραση της υπάρχουσας υπολογιστικής υποδομής σε πολλούς χρήστες, αυξάνοντας τη χρησιμοποίηση των πόρων και παράλληλα μειώνοντας το συνολικό κόστος και την ενεργειακή κατανάλωση. Οι εικονικές μηχανές (virtual machines - VMs) αποτελούν το πρώτο μέσο που χρησιμοποιήθηκε στα νέφη για την κατανομή των υπολογιστικών πόρων και τη δημιουργία απομονωμένων περιβαλλόντων μεταξύ των χρηστών. Πρόσφατα, μια πιο ελαφριά τεχνολογία εικονικοποίησης (lightweight virtualization) έκανε την εμφάνισή της στα νέφη, τα containers. Η κύρια διαφορά των containers με τα VMs, από την οποία προκύπτουν όλες οι επιμέρους διαφορές τους, είναι ότι τα containers μοιράζονται το λειτουργικό σύστημα του εξυπηρετητή στον οποίο εκτελούνται, εν αντιθέσει με τα VMs, όπου το κάθε VM εκτελεί το δικό του λειτουργικό σύστημα. Η άνοδος των υπολογιστικών νεφών συνέβαλε στη μείωση του κόστους και διευκόλυνε την αποθήκευση και επεξεργασία μεγάλου όγκου δεδομένων από πλαίσια κατανεμημένης επεξεργασίας. Πολλές πλατφόρμες νεφών, απλοποιούν ακόμη περισσότερο την εκτέλεση τέτοιου είδους διεργασιών, προσφέροντας σχετικές υπηρεσίες, εργαλεία ή ακόμη και VMs με προ-εγκατεστημένα πλαίσια κατανεμημένης επεξεργασίας. Δύο από τα πιο διαδεδομένα πλαίσια κατανεμημένης επεξεργασίας, τα οποία μελετώνται και σε αυτή τη διατριβή, είναι το Hadoop και Spark. Η παρούσα διπλωματική διατριβή αποτελείται από δύο θεματικές ενότητες. Στην πρώτη ενότητα μελετώνται, ερευνώνται και γίνεται εμπειρική αξιολόγηση των πλαισίων κατανεμημένης επεξεργασίας Hadoop και Spark. Στη δεύτερη ενότητα παρουσιάζονται και ερευνώνται καινοτόμες τεχνολογίες εικονικοποίησης, συστήνεται και αξιολογείται ο συνδυασμός αυτών και προτείνεται και αξιολογείται η υιοθέτησή τους στις σύγχρονες υπηρεσίες υπολογιστικών νεφών

Next-generation molecular diagnostics (TaqMan qPCR and ddPCR) for monitoring insecticide resistance in Bemisia tabaci

This is the author accepted manuscript. The final version is available from Wiley via the DOI in this recordData availability statement: The data that support the findings of this study are available from the corresponding author upon reasonable request.BACKGROUND: Insecticide resistance has developed in several populations of the whitefly Bemisia tabaci worldwide and threatens to compromise the efficacy of chemical control. The molecular mechanisms underpinning resistance have been characterized and markers associated with the trait have been identified, allowing the development of diagnostics for individual insects. RESULTS: TaqMan and Droplet Digital PCR (ddPCR) assays were developed and validated, in individual and pooled whitefly samples, respectively, for the following target-site mutations: the acetylcholinesterase (ace1) F331W mutation conferring organophosphate-resistance; the voltage-gated sodium channel (vgsc) mutations L925I and T929V conferring pyrethroid-resistance; and the acetyl-CoA carboxylase (acc) A2083V mutation conferring ketoenol-resistance. The ddPCR's limit of detection (LoD) was <0.2% (i.e. detection of one heterozygote whitefly in a pool of 249 wild-type individuals). The assays were applied in 11 B. tabaci field populations from four locations in Crete, Greece. The F331W mutation was detected to be fixed or close to fixation in eight of 11 B. tabaci populations, and at lower frequency in the remaining ones. The pyrethroid-resistance mutations were detected at very high frequencies. The A2083V spiromesifen resistance mutation was detected in eight of 11 populations (frequencies = 6.16-89.56%). Spiromesifen phenotypic resistance monitoring showed that the populations tested had variable levels of resistance, ranging from full susceptibility to high resistance. A strong spiromesifen-resistance phenotype-genotype (A2083V) correlation (rs  = -0.839, P = 0.002) was observed confirming the ddPCR diagnostic value. CONCLUSION: The ddPCR diagnostics developed in this study are a valuable tool to support evidence-based rational use of insecticides and resistance management strategies. © 2022 Society of Chemical Industry.European Union Horizon 2020Public Investments Project (PIP), General Secretariat for Research & Technology (GSRT)Crete Operational Program 2014-2020European Regional Development Fund (ERDF)European Social Fund (ESF

Multiple TaqMan qPCR and droplet digital PCR (ddPCR) diagnostics for pesticide resistance monitoring and management, in the major agricultural pest Tetranychus urticae

BACKGROUND Decisions on which pesticide to use in agriculture are expected to become more difficult, as the number of available chemicals is decreasing. For Tetranychus urticae (T. urticae), a major pest for which a number of candidate markers for pesticide resistance are in place, molecular diagnostics could support decision-making for the rational use of acaricides. RESULTS A suite of 12 TaqMan qPCR assays [G314D (GluCl1), G326E, I321T (GluCl3), G119S, F331W (Ace-1), H92R (PSST), L1024V, F1538I (VGSC), I1017F (CHS1), G126S, S141F, P262T (cytb)], were validated against Sanger-sequencing, and subsequently adapted for use with the ddPCR technology. The concordance correlation coefficient between the actual and ddPCR measured mutant allelic frequencies was 0.995 (95% CI = 0.991-0.998), and no systematic, proportional, or random differences were detected. The achieved Limit of Detection (LoD) was 0.1% (detection of one mutant in a background of 999 wild type mites). The ddPCR assay panel was then assessed in terms of agreement with phenotypic resistance, through a pilot application in field populations from Crete, with strong correlation and thus predictive and diagnostic value of the molecular assays in some cases (e.g., etoxazole and abamectin resistance). Molecular diagnostics were able to capture incipient resistance that was otherwise missed by phenotypic bioassays. The molecular and phenotypic resistance screening of T. urticae field populations from Crete, revealed both multi-resistant and susceptible populations. CONCLUSION The highly sensitive T. urticae molecular diagnostic platforms developed in this study could prove a valuable tool for pesticide resistance management

Programming languages for data-Intensive HPC applications:A systematic mapping study

A major challenge in modelling and simulation is the need to combine expertise in both software technologies and a given scientific domain. When High-Performance Computing (HPC) is required to solve a scientific problem, software development becomes a problematic issue. Considering the complexity of the software for HPC, it is useful to identify programming languages that can be used to alleviate this issue. Because the existing literature on the topic of HPC is very dispersed, we performed a Systematic Mapping Study (SMS) in the context of the European COST Action cHiPSet. This literature study maps characteristics of various programming languages for data-intensive HPC applications, including category, typical user profiles, effectiveness, and type of articles. We organised the SMS in two phases. In the first phase, relevant articles are identified employing an automated keyword-based search in eight digital libraries. This lead to an initial sample of 420 papers, which was then narrowed down in a second phase by human inspection of article abstracts, titles and keywords to 152 relevant articles published in the period 2006-2018. The analysis of these articles enabled us to identify 26 programming languages referred to in 33 of relevant articles. We compared the outcome of the mapping study with results of our questionnaire-based survey that involved 57 HPC experts. The mapping study and the survey revealed that the desired features of programming languages for data-intensive HPC applications are portability, performance and usability. Furthermore, we observed that the majority of the programming languages used in the context of data-intensive HPC applications are text-based general-purpose programming languages. Typically these have a steep learning curve, which makes them difficult to adopt. We believe that the outcome of this study will inspire future research and development in programming languages for data-intensive HPC applications.info:eu-repo/semantics/publishedVersio